Detecting COVID-19 from digitized ECG printouts using 1D convolutional neural networks

The COVID-19 pandemic has exposed the vulnerability of healthcare services worldwide, raising the need to develop novel tools to provide rapid and cost-effective screening and diagnosis. Clinical reports indicated that COVID-19 infection may cause cardiac injury, and electrocardiograms (ECG) may serve as a diagnostic biomarker for COVID-19. This study aims to utilize ECG signals to detect COVID-19 automatically. We propose a novel method to extract ECG signals from ECG paper records, which are then fed into one-dimensional convolution neural network (1D-CNN) to learn and diagnose the disease. To evaluate the quality of digitized signals, R peaks in the paper-based ECG images are labeled. Afterward, RR intervals calculated from each image are compared to RR intervals of the corresponding digitized signal. Experiments on the COVID-19 ECG images dataset demonstrate that the proposed digitization method is able to capture correctly the original signals, with a mean absolute error of 28.11 ms. The 1D-CNN model (SEResNet18), which is trained on the digitized ECG signals, allows to identify between individuals with COVID-19 and other subjects accurately, with classification accuracies of 98.42% and 98.50% for classifying COVID-19 vs. Normal and COVID-19 vs. other classes, respectively. Furthermore, the proposed method also achieves a high-level of performance for the multi-classification task. Our findings indicate that a deep learning system trained on digitized ECG signals can serve as a potential tool for diagnosing COVID-19.


Introduction
Since the initial infection was reported in Wuhan, China at the end of 2019, COVID-19 has become a global pandemic, infecting more than 500 million individuals and causing more than 6 million deaths (until the end of April, 2022) [1]. COVID-19 is becoming an endemic disease that will always be with us [1,2]. COVID-19 disease, when it becomes an endemic disease, may still pose problems for many people, especially vulnerable groups like the elderly and those with pre-existing medical conditions, owing to its rapid contagious [3]. As a result, classification application to evaluate its efficiency and accuracy. To the best of our knowledge, this is the first time the ECG signal extracted from ECG printouts was used to detect COVID- 19. This method allows us to normalize data from many sources, removes the model's reliance on image quality, and enhances computing performance by using 1D convolution. In addition, the image-to-signal conversion also benefits ECG data storage. The majority of ECG records were collected in paper-based form over the years [15]. However, it has the drawbacks of being damaged over time and being difficult to archive data in order to establish an electronic medical record database. A good storage effort may be a viable answer to these issues. The digitization procedure is one of the most efficient ways to save ECG data. The ECG printouts can be digitized to reduce storage space and to convey various vital morphological and functional information for clinical and diagnostic scientific study. This digital data can be then analyzed with a machine learning algorithm to detect heart abnormalities quickly. Previous studies have developed algorithms to convert paper data into 1D signals [15][16][17][18]. These algorithms, however, were either too complicated or inadequately conceived for our experimental data. In this study, we developed a simple, rapid, and effective method, based on a pixel tracking algorithm for converting an ECG image to signals. After being extracted, the 12 signal segments from 12 leads will be concatenated and fed into the SEResNet18 model [19], which will classify ECG signals of individuals with COVID-19 and other CVDs. Several scenarios were undertaken to demonstrate the capability of identifying COVID-19 based on ECG signals. In addition, we evaluated our approach by comparing it against previous studies. The findings demonstrated that our approach performs competitively and ECG could be a useful diagnostic tool for COVID-19.
In summary, our main contributions are threefold: • First, we develop a high-precision COVID-19 vs. other CVDs vs. healthy individuals classification system that takes the ECG image as direct input, then the ECG signal is extracted and fed into a 1D-CNN.
• Second, a novel algorithm for converting ECG paper records to digital signals was developed, which took ECG images as input and converted them into electronic ECG recording forms.
• Third, extracted 1D ECG signals are used as input for the machine learning model in the COVID-19 diagnosis task for the first time, to the best of our knowledge. Extracting ECG signals from images provides the following advantages: (i) computational cost savings due to the use of 1D-CNN; (ii) application in converting ECG paper into soft files for archiving; (iii) hand-crafted feature extraction methods can be applied to ECG signals for further analysis.
• Last, by simulating the data settings of prior research and conducting extensive experiments, we demonstrate that models using ECG signal as input outperform the current state-of-theart, which does the classification directly on the ECG image. Our codes for signal-to-image conversion and modeling are made available at https://github.com/vinuni/DiECG-Cov.
This paper is structured as following. In Methods section, the image-to-signal conversion method is explained. In section Experiments, the used dataset is presented first, followed by a description of the experimental settings and evaluation criteria that we employ to assess model performance. In Results section, the experimental results are presented and analyzed. The main findings and limitations of the study are summarized and some useful suggestions are provided in Discussions. Finally, the entire study are summarized in the Conclusion section.

Methods
Our proposed approach for identifying COVID-19 from ECG images involves two steps: converting images into 1D signals and feeding these signals into a CNN model that distinguishes between COVID-19, other CVDs, and normal ECG. The schematic diagram for the proposed system is illustrated in Fig 1.

Image-to-signal conversion algorithm
ECG data is often stored in paper form, which consumes storage space, and as a machine learning model input, 2D image data requires more computational cost than time series data (1D signal). For those reasons, we extract the 1D ECG signal from the scanned ECG paper and use that signal as input to the classifier. The procedure for extracting the ECG signal consists of 5 steps as depicted in Fig 2. First, the scanned image is converted to a binary image. From the binary image, the isoelectric lines of the ECG signal are found, and from the position of these lines, the image of each lead is separated. Since the binary image contains noise and

PLOS ONE
Detecting COVID-19 from digitized ECG printouts using 1D convolutional neural networks artifacts, a noise removal algorithm is required. Finally, from the signal image of each lead, the ECG signal is extracted and concatenated to form a time series ECG signal.
Image binarization. ECG scans are converted from colored to binary by Otsu's method [20] to facilitate further processing.
Isoelectric lines detection. It is critical to find the position of the isoelectric lines in the ECG recording, which can deduce the area of each lead on the image. Due to the characteristic of the isoelectric lines, when projecting the ECG image in the vertical direction, the data points (black points as shown in Fig 3a) will be concentrated with high intensity at their position. As shown in Fig 3b, the vertical histogram of the image has peaks at the positions of the isoelectric lines. Thus, finding the position of the isoelectric lines can be done by finding the peaks of the histogram. As shown in Fig 3d, the isoelectric lines are detected (the black horizontal lines) and shared by four signals corresponding to four ECG leads each.
Lead seperation. From the position of the detected isoelectric lines, we separate the image of each lead by zoning vertically from the upper isoelectric line minus L pixels to the lower isoelectric line plus L pixels (L depends on the characteristics of ECG images, specifically the Noise and artifacts removal. The binary image is embedded in noise and artifacts from the grid and adjacent signals (as shown in Fig 4b). We built a simple yet effective algorithm that separates the image of the ECG signal from noise and artifacts. The first step involves initializing a point that belongs to the isoelectric line and has a pixel value of 0. This point has coordinate (n, b) (b is the y-intercept of the isoeletric line). Pixels satisfying the following three conditions i) located on the same column as the starting point; ii) having a value of 0 and iii) less than N pixels away from the starting point; are retained. Pixels with a value of 0 in the n th column of the image and are less than N pixels away from the isoelectric line are considered pixels of the ECG signal in that column. Similar to L, the value of N depends on the characteristics of ECG images, specifically the thickness of ECG lines. The maximum coordinate (c max ) and the minimum coordinate (c min ) of pixels 0 in that column are calculated. In the next column, pixels with coordinates from c min − N to c max + N are considered; and among them, pixels with value 0 are retained. c max and c min values of this column are again calculated, and the search for ECG pixels in the next column is carried out.
From the starting point, we track in two directions: to the left edge of the image and to the right edge of the image. At the end of the process, only ECG pixels are retained. The noise removal algorithm is detailed in Algorithm 1 and  Signal extraction. The image containing only the ECG pixels is the subject of the imageto-signal conversion.
Extraction of 1D ECG signals from images is conducted by scanning the image horizontally. The amplitude of the ECG signal at a specific column is equal to the average of the pixel 0 positions of that column minus the isoelectric line position. For columns missing pixel 0, the signal amplitude is taken as the value of the signal point immediately preceding it. Once the scanning is done, the length of the resulting signal is equal to the number of image's columns.
Each ECG printout has reference pulses in the form of a square pulse (as shown in Fig 6). In Fig 6a, 6b and 6d, the reference square pulses have a width corresponding to 0.1 s and a

PLOS ONE
Detecting COVID-19 from digitized ECG printouts using 1D convolutional neural networks height corresponding to 0.5 mV. In Fig 6c the reference pulse is 0.2 s and 0.5 mV for width and height, respectively. Determining the width and height of these square pulses makes the image-to-signal conversion consistent in time and amplitude. In addition, the scale conversion also helps to display the signal accurately in terms of amplitude and time. In this study, we convert all the records to the same sampling frequency of 200 Hz. Signal quality assessment. Once the 1D ECG data have been obtained, we assess its quality to guarantee that the image-to-signal conversion algorithm functions properly and that the resulting signal can be used as input for the machine learning model. After randomly selecting 50 samples from each category, we hand-labeled the R peaks of the lead II signal in these images and regarded the resulting positions as ground truths to evaluate our algorithm against. The selected samples are also fed into our proposed image-to-signal conversion, and the generated signals were exposed to NeuroKit2's [21] R peak detection algorithm to obtain the positions of R peaks. Finally, we evaluate our algorithm by comparing its output against the ground truths, specifically by quantifying the difference in the RR intervals (the time elapsed between two successive R-waves) using the mean absolute error (MAE) metric.

Experimental setups
Dataset. COVID-19 ECG images dataset-a publicly available dataset of ECG images of cardiac and COVID-19 patients was used in this study. Khan et al. from the University of Management and Technology shared the dataset to Mendeley Data [11]. The dataset contains photos of 1937 paper-based ECG reports. ECG records for 250 COVID-19 patients, 77 MI patients, 548 individuals with irregular heartbeats (recovered from COVID-19 or MI), 203 patients with MI history (RMI), and 859 people without any cardiac abnormalities were analyzed by experts. The provided dataset is the first ECG dataset for COVID-19 disease that has been shared and is completely anonymized. We did not seek approval from an ethics committee and participants' consent for this study because (i) the previous study during which the dataset was collected already obtained necessary approval, (ii) the dataset is completely anonymized and contains no personal information which can be used to identify patients, and (iii) no further data are collected during this study.
The ECG images in the dataset are from a 12-lead system (I, II, III, aVR, aVL, aVF, V1, V2, V3, V4, V5, and V6) at a sampling rate of 500 Hz. ECG data were gathered using an EDAN SE-3 series 3-channel electrocardiograph, and some of the signals were applied with a 0.67-25 Hz bandpass filter, while others were processed with a 0.5-100 Hz bandpass filter and a 50 Hz notch filter, according to the paper-based ECG reports.
The dataset has some limitations. In particular, the dataset is considerably imbalanced, with 859 samples of normal ECG (the class with the most samples) and 77 samples of MI ECG (the class with the fewest samples). Besides, the dataset consists of four different types of reports, which are different in (i) image size and resolution; (ii) ECG report template; (iii) grid background, and (iv) reference pulse. Four types of ECG printout templates are shown in Fig 6. Using the signal extracted from the ECG image is not only reduces the computational cost but also removes disparities in ECG printout templates, making the prediction more objective.
Experiments. To evaluate the capability of diagnosing COVID-19 based on ECG data, three classification tasks are covered: (i) COVID-19 and normal ECG; (ii) COVID-19 and others; (iii) COVID-19, normal and others. As the number of samples in MI and RMI classes is relatively small, the merging of MI, RMI, and abnormal into a class could alleviate the effects of data imbalance. The number of images used in each problem is detailed in Table 3.
To evaluate the effectiveness of this method, we also compare its performance against the results of previous studies performed on the COVID-19 ECG images dataset. We simulate the data splitting of previous studies and execute similar classification tasks [12][13][14].
Implementation details. In this study, all experiments were conducted using Intel core i9-10900X CPU @ 3.70GHz, RAM 31 GB hardware and NVIDIA GeForce RTX 3080 Ti GPU. We trained 110 epochs for each model, with a mini-batch size of 16, and evaluated model performance after each epoch. Checkpoint with the highest F1-score against the test set was considered the best model for each training procedure. As the backbone of the COVID-19 vs. Normal classifier, we apply multiple architectures, then select the model with the best performance to train other classification tasks. The utilized architectures are SEResNet18 [19], ResNet18 [22], ResCNN [23], Sequence Stroke Net [24], LSTM Fully Convolutional Network (LSTM FCN) [25], GRU Fully Convolutional Network (GRU FCN) [26], InceptionTime [27], XceptionTime [28] and Time Series Transformer (TST) [29]. We adopted Adam optimizer [30] (β1 = 0.9, β2 = 0.999 and learning rate = 1e-3), cooperating with Cosine annealing learning rate [31] with the maximum number of iteration set to 25 and the learning rate stop decaying at 100 th epoch. As a criterion for training, the label smoothing cross-entropy loss function with a smoothing factor of 0.1 was used. We selected the most suitable hyper-parameters using simple grid search strategy and experimental results.
To evaluate the performance of the models, a nested cross-validation method was employed for each trial. The dataset was first divided into 5 folds, with 1 set used for testing and the rest used for training and validation. In each iteration, we used 4-fold cross-validation to further split the data between training and validation. Classification performance was evaluated for each fold, and then the average classification performance of each model was calculated. To ensure that the data distributions in training, validation and test sets were the same as in the original dataset, stratified splitting was adopted for both cross-validation steps.
We used all of the samples in the dataset that are relevant to the considered classes for each classification task. The number of images used for each task is listed in Table 3.

Evaluation metrics
To evaluate the quality of the ECG signals extracted from the images, we used mean absolute error (MAE) as the evaluation metric. The mean of the RR intervals in the ground truth and the resulting signal is calculated, then MAE value is calculated. Eq 1 gives the formula for computing the MAE.
Where N is total number of ECG images; mRR i is the mean of RR intervals calculated from i th image;m RR i is the mean of RR intervals of the corresponding signal.
In terms of the CNN model, model performance is assessed utilizing well-known evaluation metrics such as Accuracy, F1-score, Specificity, Sensitivity, and AUC (Area Under the Curve of the Receiver Operating Characteristic).

Experimental results and implications
ECG signal quality assessment. Extracted signal quality assessment was evaluated on a certain number of samples in the dataset. The number of samples in each class, as well as the MAE of the RR interval, are listed in Table 1. Despite starting with 50 samples in each class, the RR interval cannot be calculated for samples with only one R peak and thus the reported sample counts were less than 50.
The MAE values range from 11.09 ms to 63.75 ms, as shown in Table 1. Normal class has the lowest MAE (11.09 ms) due to the regular and stable ECG waveform. Class MI has the highest MAE of 63.75 ms, due to the unstable of ECG form of patients with MI disease. The MAE values for COVID-19, Abnormal, and RMI classes are 16.07 ms, 25.73 ms, and 27.57 ms, respectively. Overall, the mean MAE of the tested samples is 28.11 ms, which is 3.7% of the RR interval for a person with an 80 bpm heart rate (equivalent to a mean RR interval of 750 ms). ECG signal classification. Table 2 compares the COVID-19 vs. normal classification results between different models. With an f1-score of 0.9842 and an AUC of 0.9735, the SERes-Net18 model outperforms the others. The XceptionTime model has the best classification accuracy (98.58%), while the ResNet18 model has the fastest inference time (18.34 ms per sample). According to the obtained results, we choose the SEResNet18 with some modifications as the backbone for other classification tasks. Convolution (Conv) layers are modified with a larger kernel size (kernel_size = 7) and a stride parameter of 1 in order to capture longer patterns in ECG data. For ECG data in specific and time-series data in general, this technique has been recommended to be more effective [27,32].  Table 3 covers all of the experimental results for each classification task. Patients positive for COVID-19 were distinguished from normal patients and patients negative for COVID-19 with high accuracy of 98.42% and 98.50%, respectively. The classification accuracy and AUC were 83.17% and 0.8748 in the case of classifying patients with COVID-19, Normal and remaining classes. The majority of misclassified samples include COVID-19 ECGs with identical characteristics to those of normal ECGs, indicating that the ECG waveform of the patients has not changed.
It is also crucial to discover whether the network is making the decision based on the relevant segment of the ECG signal or somewhere else. For different classes of ECG data, heat maps based on the Grad-CAM technique [33] were constructed. Fig 8 shows heat maps generated with the best-performing models and Fig 9 depicts t-SNE plots [34] to visually highlight the capacity to distinguish between COVID-19 and other conditions. Table 4 summarizes the experimental results and comparisons against previous studies using the same ECG dataset. The proposed method has an accuracy of 0.6% higher than that of previous studies in the COVID-19 vs. Normal classification problem and up to 6.72% and 6.43% higher in the other classification problems in terms of to accuracy and F1-score, respectively.

Discussions
In addition to having hazardous effects on human health, the COVID-19 pandemic has placed a significant strain on the healthcare system. To alleviate this problem, a rapid and straightforward method for COVID-19 diagnosis is required. The use of a deep learning methodology using 1D signals converted from ECG images to detect COVID-19 shows potential as a novel diagnostic procedure. When compared against previous works on the same dataset, our method performed competitively. The following reasons may contribute to the proposed method's better performance: (i) white pixels contain no information, although they account for a major portion of the image; (ii) the amount of data required to train a 2D CNN is greater than that of 1D CNN [35] while the experimental dataset contains a modest number of samples (250 images for COVID-19 class), and (iii) it is thought that for ECG signal processing, a CNN with a larger kernel size and a lower stride value (compared to values often used in image processing) would be more effective in capturing the temporal correlation of adjacent signal points [32].
This study contributes to the literature in numerous ways in this regard. These contributions can be summarized as follows: • Our work, together with previous research, demonstrated the capability to automatically distinguish COVID-19 patients by ECG signals with high sensitivity, cost-effectiveness, and non-invasiveness.
• To classify paper-based ECG data, a new and effective technique was developed, in which signals from a standard 12-lead ECG are extracted from the image and utilized as input to a 1D CNN model. This is the first time the ECG signal extracted from ECG printouts was used to detect COVID-19, to the best of our knowledge.  There are some drawbacks to our proposed method. Firstly, similar to all current imagesignal conversion algorithms, our proposed algorithm is sensitive to the shape of the ECG signal. The algorithm cannot differentiate the ECG signal in different leads if their amplitude is too high and combine together (signals with excessive amplitude are caused by noise or abnormal electrical activity of the heart due to CVDs). Furthermore, although the proposed method has been tested with variety of experimental scenarios, it still has to be evaluated with different datasets. In future works, we expect to integrate a COVID-19 diagnosis module into a mobile application, which will help medical practitioners by providing a quick and accurate approach to identifying COVID-19. Finally, although our method only produced the final diagnosis, an additional output that highlights the anomalies in the ECG [36] of COVID-19 patients can also be helpful to cardiologists. Other potential approaches will be considered include training deep learning models with hierarchical disease dependencies [37] or demographic data [38].

Conclusion
In this work, a novel and effective method for automatically detecting COVID-19 utilizing ECG printouts is proposed. The proposed approach is based on an image-to-signal conversion algorithm, which extracts 1D ECG signal from scanned ECG printouts and feeds it into the 1D SEResNet18 model. We have run experiments on various scenarios and analyzed the findings. We also simulated the data set-up of prior research to develop classification models, then compared the performance of those models with their findings to evaluate the effectiveness of utilizing the extracted signal. The findings suggest that the ECG signal may be used to distinguish COVID-19 patients from other groups, and the proposed technique outperforms previous studies using the same dataset. Additionally, the capacity of the proposed method to distinguish COVID-19 ECG can serve as evidence that there are changes in ECG signals of COVID-19 patients. The proposed ECG-based COVID-19 diagnose is easily extended to real-time systems and executed on mobile applications in our future works. In addition, we also intend to further interpret the model's decision and discover how COVID-19 patients' ECG changes compared to healthy people. As a result, it may benefit healthcare providers by offering a quick and accurate way to detect COVID-19, as well as reducing hospital costs by avoiding unnecessary hospital visits.